A virtual tour (or virtual reality tour) is a virtual reality simulation of an existing location, which is usually built using contents consisting principally of 2D panoramic images, sequence of linked still images or video sequences, and/or image-based rendering (IBR) consisting of image-based models of existing physical locations, as well as other multimedia content such as sound effects, music, narration, and text. A virtual tour is accessed on a personal computer (typically connected to the Internet) or a mobile terminal. Although not replacing real travel, virtual tours aim at evoking an experience of moving through the represented space. Virtual tours can be especially useful for universities and in the real estate industry, looking to attract prospective students and tenants/buyers, respectively, eliminating for the consumer the cost of travel to numerous individual locations.
The word panorama indicates an unbroken view, so essentially, a panorama in that respect could be either a series of photographic images or panning video footage. However, the terms ‘panoramic tour’ and ‘virtual tour’ are generally associated with virtual tours created using stills cameras. Such virtual tours created with still cameras are made up of a number of images taken from a single view point. The camera and lens are rotated around what is referred to as a nodal point (the exact point at the back of the lens where the light converges). These images are stitched together using specialist software to create a panorama representing a near 360 degree viewing angle, as viewed from a single “view point”; the panoramas are each resized and configured for optimal on-line use. Some ‘panographers’ will then add navigation features, such as hotspots (allowing the user to “jump” from one viewpoint or panorama to the next) and integrate geographic information such as plans or maps.
Current virtual tours photographic techniques suffer from several limitations. A seamless panoramic image can not be created from still images whenever such still images are captured from different nodal points or, for two consecutive images, from a single nodal point but with different focal lengths or focus distances. Images captured from a single camera rotating on its nodal point can be stitched seamlessly but this solution can not be used for applications involving axial translation, where, for example, images are captured from a vehicle in motion.
Catadioptric optical systems provide images having a 360° horizontal field of view and near 180′ vertical field of view. The resulting panoramic images are of a annular shape and generally must be sliced and open and “unwarped” to create a panoramic image of a rectangular shape. The unwarping step causes image distortion which, together with the optical distortions caused by the Catadioptric optics having unevenly distributed angles along its radial axis (vertical axis of the view), must be compensated by specialised application software.
Patent document US 2007/0211955 to Pan discloses a perspective correction method allowing e-panning without image distortion, wherein image correction step is performed on image slices (horizontal section of the wide-angle image) by repositioning each pixel to a corresponding point on a cylindrical surface. This method consumes significant processing power and bandwidth for respectively correcting and transmitting the images whenever fast user motion is involved during navigation, and is therefore not optimal for providing a seamless navigation experience at relatively high user directed panning speed.
Also, with current image capture solutions, objects near the camera are responsible for occlusion on distant objects; “occlusion” meaning, with regard to 2D images, the non-projection of a surface to a point of observation, and with regard to a 3-D space, the effect of one object blocking another object from view. Limitations of current virtual tour technology, such as object occlusion, have had the detrimental result on virtual tours never materializing outside of the real estate industry.
Virtual Walk-Through (“VWT”) applications constitute an evolution over virtual tours. This technology eliminates the occlusion limitation by enabling user to travel to a point where distant objects are no longer occluded.
Commercial online walk-through products such as Google “STREETVIEW” provide virtual outdoor walk-through of cities using images captured from a camera mounted on a road vehicle which circulates on motorways and roads at speeds ranging from 30 kmh to 80 kmh. These products are limited to outdoor views wherein any two consecutive points of view are positioned at a relatively long distance from each other. Typically, “STREETVIEW” and applications of the like provide visualisation at road-view level, that is, visiting a city as viewed from a car, wherein the user follows a pathway formed by a plurality of panoramas accessible along main streets, the user following the pathway by “jumping” (generally from a graphical interface allowing clicking of on-screen icons) from a panorama or point of view to the next distant point of view. Google “STREETVIEW” also creates ground plane distortion where planar ground seems to be inclined due to the unwanted motion of the cameras caused by the inertial force. Other current walk-through products “EARTHMINE” (www.earthmine.com by Earthmine Inc., Berkeley, Calif.) provides commercial online walk-through for applications such as management of buildings and assets, telemetric measurement and other cadastral works. The product combines high resolution still images and 3D mesh information to provide pathways wherein a user jumps from one distant view point to another. “EVERYSCAPE” (www.everyscape.com by Everyscape, Waltham, Mass.) provides commercial online panoramic products wherein motion between two consecutive view points is simulated by video postproduction effects. This product does not allow the user to pan and tilt his viewing angle during its displacement along the travel path. During motion, images are no longer panoramic unless the fields of view of images representative of the next fixed point, are constrained to the motion axis.
The prior art covers several techniques that aim at reducing the bandwidth associated with the transmission of panoramic images and applications between a server and the user's remote terminal, allowing a user to navigate a walk-through space while downloading data.
The use of a predefined pathway has been widely adopted to prevent storage and transmission of redundant image data. Predefined pathways have the additional benefit of simplifying user navigation, notably by preventing a user from searching available paths or from hitting objects repetitively during motion, as would he the case when user tries to walk through walls or door images.
U.S. Pat. Nos. 6,388,688 and 6,580,441, both to Schileru-Rey, disclose a computer system and method that allow interactive navigation and exploration of spatial environments, wherein pathways are represented by branches, and intersections in the real environment are represented by nodes. User selects which path to follow from the node. A branch represents a video sequence or animation played during motion between two adjacent view points. Virtual objects can be integrated to specific branches or nodes, without assigning a geographic coordinate to virtual objects; each object is linked to at least one branch or node and displayed when user is travelling on said branch or node.
U.S. Pat. No. 6,968,973 to Uyttendaele discloses a system for providing an interactive video tour, wherein the order of frames to be displayed is determined based on the user-specified direction of motion along the tour path. Scene data includes a branch points table which for each video frame associated with a branch point, identifies the next video frame that is to be played in view of the current viewing direction. Patent 973' does not provide a device to speed up downloading of frames, beyond selection of the next frame, and thus do not permit downloading of data based on anticipated user motion.
U.S. Pat. No. 7,103,232 to Kotake discloses an IBR system with improved broadcasting performance, where a panoramic image is created by stitching several images from several cameras, preferably video cameras, pointing to distinct points of view, the cameras being synchronised by use of a single time code. The '232 system provides panoramic images divided in six images sections of 60° horizontal field of view, and broadcast typically only two of the six sections' image (providing a 120° field of view) at any giving time with an aim to reduce processing power and communication bandwidth. The '232 solution is not optimized however for walk through applications allowing fast movement across the horizontal plane beyond 120°; moreover, the '232 patent does not disclose broadcasting images of different image resolution, meaning that it only covers broadcasting of images of the highest possible image resolution.
U.S. Pat. No. 6,633,317 to Jiang Li discloses a data transfer scheme, dubbed spatial video streaming, allowing a client to selectively retrieve image segments associated with the viewer's current viewpoint and viewing direction, rather than transmitting the image data in the typical frame-by-frame manner. The method of patent 317' divides walk-through space in a grid. Each cell of the grid is assigned to at least one image of the surrounding scene as viewed from that cell. Images may be characterised as a concentric mosaic in that each cell is represented by a sequence of image columns. The method of patent 317' allows transmission of part of the images (compressed or not) needed in an attempt to anticipate the viewer's change of view point within the walkthrough space, starting with image data corresponding to viewpoints immediately adjacent the current viewpoint, with subsequent image data associated with viewpoints radiating progressively out from the current viewpoint. Patent 317' is well suited to open walkthrough spaces where user can move in any direction using multiple sources of image data (simple 2D images, panoramic images or concentric mosaics), such as in a typical 3D environment. However, this method is not suited to the optimal transmission of full panoramic images in situations where the user travels along predefined pathways consisting of several view points in a linear arrangement within a network of pathways within the walkthrough space. Additionally, being view direction sensitive, this method is not optimized in terms of response time so as to allow the user to change his travel plan, for example by making a U-turn or travelling along another pathway. Finally, as this method allows travel in any direction (along predefined pathway), the amount of data download to represent a given view point is greater and therefore less suited for a fast and responsive viewing experience on the Internet or other network media having limited bandwidth.
Consequently, no system of the prior art provides a system optimized for seamless broadcasting of fluid motion where the user can orientate (pan and tilt) the field of view during motion and where the user can stop the motion anywhere along the travel path, in order to discover local objects in detail without occlusion.
U.S. Pat. No. 6,693,649 to Lipscomb discloses a solution for non-linear mapping between media and display allowing “hotspots”, defined as an outline of two points connected by straight lines, to be used in the context of panoramas. Such “hotspots” are referenced to each image using two angles or two pixel coordinates and as such, these values are only valid for each distinct image. Patent 649' does not use geographic coordinates based on 3D with an x, y and z axis. Allocation of three dimension coordinates for each virtual objects and determination of precise geographic location information for each view point, according to embodiments of the present invention, are prerequisite for the seamless integration of hotspots and other virtual objects in panoramas, where these objects would be visible from any point having a direct sight line to the object.
Consequently, no prior art system provides a system and advanced features based on geographical information such as the ability to pin an element of information on any location in a view, such element staying spatially fixed to the point during the travel.
The present method and system for creating and broadcasting interactive walkthrough applications overcomes these shortcomings as will now be described.
Given the market need for immersive walkthrough applications, what is needed therefore is a system and method for the real-time assembly and broadcasting of walk-through applications using any combination of 2D panoramic images and virtual 3D images, with optionally virtual objects, that can provide seamless, quality, fluid walk-through navigation.
What is needed is an optimized system or method for the real-time assembly and broadcasting of panoramic walkthrough applications providing immersive, interactive and intuitive navigation experience, which allow the user, from each view point or geographical coordinate along a network of pathways, to have a complete view from a first person point of view, the view covering substantially 360° in field of view.
What is needed is a system or method that allows optimising broadcast and real-time construction of panoramic walk-troughs applications, providing ongoing image broadcasting to ensure seamless user motion, wherein the combination of images download anticipates user travel, based on the ability of the user to choose the direction of displacement at each intersection point.
Further, what is needed is a system or method for broadcasting and real-time construction of panoramic applications, wherein the balance between response time (possibility to navigate on different pathways) and fluidity of movement and resolution (characterized by the number of different images available along a given pathway) is optimised.
Finally, what is needed is a system or method for creating and broadcasting interactive panoramic walk-through applications that can provide genuinely interactive functions accessible from the images.